Calculating conjunctival redness

ABSTRACT

The present application includes methods, systems and computer readable storage devices for determining a color score for at least a portion of a biological tissue. The subject matter of the application is embodied in a method that includes obtaining a digital image of the biological tissue, and receiving a selection of a portion of the image as an evaluation area. The method also includes determining for each of a plurality of pixels within the evaluation area, a plurality of color components that are based on a Cartesian color space, and determining, from the color components, a hue value in a polar coordinate based color space. The method further includes determining a color value based on the hue value for each of the plurality of pixels, and assigning a color score to the evaluation area based on an average of the color values of the plurality of pixels.

CROSS REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. application Ser. No.13/773,310, filed Feb. 21, 2013, which claims priority to U.S.provisional application No. 61/601,484, filed on Feb. 21, 2012, theentire content of which is incorporated herein by reference.

FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

This invention was made with Government support under Grant No. K24EY019098 awarded by the National Institutes of Health. The Governmenthas certain rights in the invention.

TECHNICAL FIELD

This disclosure relates to quantitative determination of a particularcolor content in an image or a portion of the image.

BACKGROUND

Colored images are often used in medical applications. For example,conjunctival redness has important value in the evaluation of ocularinflammatory or surface disease. Conjunctival redness can be assessedsubjectively based on a physician's clinical criteria. Clinicians canguide their criteria using sets of photographs, of which there aredifferent versions and none is considered as a standard. Evaluatingconjunctival redness by simple visual observation, whether based onimages or direct in-person observation, introduces subjectivity and thusa potential source of bias.

SUMMARY

Described herein are methods and systems that facilitate computer basedevaluation of colors in digital images. The digital images can includeclinical images or photographs of patients. Selected portions of imagestaken with different imaging systems can be evaluated with respect to astandardized score that represents specific color content in theselected portions. For example, a portion of a digital image of an eyecan be selected to determine a redness score of the selected portion.Such a redness score can be referred to as a Conjunctival Redness Index(CRI) or Ocular Redness Index (ORI). Conjunctival redness can be closelyrelated to ocular redness, and therefore the term CRI may also beinterchangeably used with the term Ocular Redness Index (ORI).

In one aspect, this application features a computer implemented methodfor determining a color score for at least a portion of a biologicaltissue. The method includes obtaining a digital image of the biologicaltissue, and receiving a selection of a portion of the image as anevaluation area. The method also includes determining for each of aplurality of pixels within the evaluation area, a plurality of colorcomponents that are based on a Cartesian color space, and determining,from the color components, a hue value in a polar coordinate based colorspace. The method further includes determining a color value based onthe hue value for each of the plurality of pixels, and assigning a colorscore to the evaluation area based on an average of the color values ofthe plurality of pixels.

In another aspect, the application features a system for determining acolor score for at least a portion of a biological tissue. The systemincludes an imaging system configured to obtain a digital image of thebiological tissue, and a color score calculator module. The color scorecalculator module is configured to receive a selection of a portion ofthe image as an evaluation area through a user interface. The colorscore calculator module is also configured to determine, for each of aplurality of pixels within the evaluation area, a plurality of colorcomponents that are based on a Cartesian color space, and determine,from the color components, a hue value in a polar coordinate based colorspace. The color score calculator module is further configured todetermine a color value based on the hue value for each of the pluralityof pixels and assign a color score to the evaluation area based on anaverage of the color values of the plurality of pixels.

In another aspect, the application features a computer readable storagedevice having encoded thereon computer readable instructions, which whenexecuted by a processor, cause a processor to perform severaloperations. The operations include obtaining a digital image of thebiological tissue, and receiving a selection of a portion of the imageas an evaluation area. The operations also include determining for eachof a plurality of pixels within the evaluation area, a plurality ofcolor components that are based on a Cartesian color space, anddetermining, from the color components, a hue value in a polarcoordinate based color space. The operations further include determininga color value based on the hue value for each of the plurality ofpixels, and assigning a color score to the evaluation area based on anaverage of the color values of the plurality of pixels.

In another aspect, the application features a method of determiningseverity of an eye condition, wherein increasing severity of thecondition is associated with increasing ocular redness. The methodincludes determining a subject color score for an ocular tissue, whereinthe subject color score as compared to a reference color score indicatesthe severity of the condition. For example, the presence of a subjectcolor score that is above a reference color score indicates that thesubject has a severe disease.

In another aspect, the application features a method of monitoring theefficacy of a treatment for an ocular condition in a subject. The methodincludes determining a first subject color score for an ocular tissueand administering one or more treatments to the ocular tissue. Themethod also includes determining a second subject color score for theocular tissue; wherein the first and second subject color scores aredetermined as described herein. The first and second subject colorscores are compared, wherein a decrease in the color scores from thefirst to the second indicates that the treatment was effective, and nochange or an increase in the color scores indicates that the treatmentwas ineffective.

In another aspect, a method of monitoring the progression of an ocularcondition in the subject includes determining a first subject colorscore for an ocular tissue, and determining a second subject color scorefor the ocular tissue. The first and second subject color scores aredetermined as described herein. The method also includes comparing thefirst and second subject color scores, wherein a decrease in the colorscores from the first to the second indicates that the condition isimproving, no change in the color scores indicates that the condition isstable, and an increase in the color scores indicates that the conditionis worsening.

Implementations can include one or more of the following.

The digital image can include an area of reference white color. Aselection of at least a portion of the reference white color can bereceived and an average gain associated with the portion of thereference white color can be determined. The average gain can be appliedto each of the plurality of pixels within the evaluation area.Determining the color value can include mapping an angle correspondingto the hue value to a scalar value within a predetermined range, anddetermining the color value as a product of the scalar value and atleast one component of the polar coordinate based color space that isdifferent from the hue. A parabolic curve can be used in mapping theangle to the scalar value within the predetermined range. The selectionof the evaluation area can be received through a graphical userinterface in which the digital image is presented. The selection of theevaluation area can be received as a polygonal area of the digital imageselected through the graphical user interface. The color score canindicate a degree of redness of the biological tissue. The biologicaltissue can be an ocular tissue or an epidermal tissue. The ocular tissuecan be conjunctiva. The epidermal tissue can be facial epidermis. TheCartesian color space can be an RGB color space. The polar coordinatebased color space can be an HSV color space. The color score and anassociation of the color score with the digital image can be stored in astorage device. The digital images can be related to one or more ofcorneal neovascularization, fluorescein stained epithelial punctatekeratitis and epithelial defects, eyelid and skin telangiectasia, andconjunctival/scleral pigmented lesions.

The user interface can be a graphical user interface in which thedigital image is presented. The graphical user interface provides adigital selector for receiving the selection of the evaluation area as apolygonal area in the digital image. The condition can be selected fromthe group consisting of dry eye syndrome, conjunctivitis,subconjunctival hemorrhage, blepharitis, acute glaucoma, allergy,injury, keratitis, iritis, episcleritis, scleritis, uveitis, inflamedpterygium, inflamed pinguecula, airborne contaminants, Rocky Mountainspotted fever, high stress levels and drug use including cannabis.

Particular implementations may realize none, one or more of thefollowing advantages. Images taken using different imaging systems canbe evaluated based on a standardized scale and quantitatively comparedwith one another. For example, images taken across different imagingsystems, lighting conditions, or time can be subjectively compared withone another. This allows for repeatability, consistency and accuracybecause the scoring does not rely on the subjective judgment of a humanobserver, requiring only a personal computer and an operator with basictraining on computer use. Evaluation of the images does not require anyspecial training. For example, evaluating conjunctival redness in an eyeimage using the methods and systems described herein does not requiretraining in ophthalmology. The methods and systems described herein thusprovide an objective common denominator that can be used, e.g., todiagnose and to monitor response to therapy, using a computer basedanalysis. The methods can be used simultaneously in different clinicalsettings with minimal requirements of technological infrastructure. Byallowing for a selection of a custom evaluation area in a digital image,specific areas, e.g., areas including small lesions or the tarsalconjunctiva in an eye image, within the image can be evaluated.

Unless otherwise defined, all technical and scientific terms used hereinhave the same meaning as commonly understood by one of ordinary skill inthe art to which this invention belongs. Methods and materials aredescribed herein for use in the present invention; other, suitablemethods and materials known in the art can also be used. The materials,methods, and examples are illustrative only and not intended to belimiting. All publications, patent applications, patents, sequences,database entries, and other references mentioned herein are incorporatedby reference in their entirety. In case of conflict, the presentspecification, including definitions, will control.

Other features and advantages of the invention will be apparent from thefollowing detailed description and figures, and from the claims.

DESCRIPTION OF DRAWINGS

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawing(s) will be provided by the Office upon request and paymentof the necessary fee.

FIG. 1 is a block diagram of an example of a system for calculatingcolor scores of images acquired through different imaging systems.

FIG. 2A depicts a Cartesian color space.

FIGS. 2B-2D depict polar coordinate based color spaces.

FIG. 3A is a flowchart representing an example sequence of operationsfor determining a color score of a digital image.

FIG. 3B is an example of a curve used in an operation depicted in FIG.3A.

FIGS. 3C and 3D show two images before and after white balancing,respectively.

FIGS. 3E-3L illustrate examples of white balancing using a marker.

FIG. 4 is a flowchart representing example sequence of operations fornormalizing gain in a digital image.

FIG. 5 is a block diagram of an example of a computing system.

FIGS. 6A-6C are examples of images in which color scores have beencalculated.

FIGS. 7A-7D show results illustrating a relationship between calculatedredness scores and clinician evaluations.

DETAILED DESCRIPTION

Evaluating a color content of a digital image has various applications.In some cases, diagnostic or therapeutic procedures can rely onevaluating digital images representing patient conditions. As oneexample, evaluating conjunctival redness from a digital image of apatient's eye can be important in ophthalmologic applications.Conjunctival redness, also known as bulbar hyperemia, erythema, ocularhyperemia, redness, or injection, is associated with an increaseddilation of blood vessels in the conjunctiva, and is a common clinicalsign in subjects suffering from a wide range of ocular conditions, suchas irritation or an inflammatory response, e.g., due to contact lenswear, medications, or pollutants. In such cases, conjunctival rednesscan be a useful indicator that can be used, for example, in diagnosis aswell as tracking response to treatment.

In some cases, conjunctival redness can be visually evaluated by anophthalmologist. However, such determination can be inaccurate,subjective and prone to human errors such as bias and inconsistency.Further, images acquired with different imaging systems cannot bereliably compared with one another due to, for example, innatevariability across different systems, lighting conditions or time.

Described herein are methods and systems that allow computer basedanalysis of digital images such that images across different imagingsystems, conditions and time can be evaluated based on a standardizedscore assigned to each of the images. The score assigned to each of theimages is based on a particular color content in the image or in aportion thereof. For example, the eye images described above can beevaluated to assign a redness score (referred to herein as aConjunctival Redness Index (CRI)). The score is calculated such thatvariability across different imaging systems, conditions and time isaccounted for. Therefore, the calculated score serves as a basis forcomparing images acquired from various sources. In addition, the methodsand systems described herein provide the flexibility of selecting aparticular portion of the image to be evaluated.

While this disclosure uses examples of evaluating redness of eye images,the methods and systems described herein can also be used to measurecolors in other tissues, e.g., to evaluate corneal neovascularization,fluorescein stained epithelial punctate keratitis and epithelialdefects, eyelid and skin telangiectasia (e.g. as in rosacea), andconjunctival/scleral pigmented lesions. Other colors, such asyellowness, can also be measured, e.g., to determine levels of jaundiceor other pathological conditions.

FIG. 1 shows a block diagram of an example of a system 100 forcalculating color scores of images acquired through different imagingsystems. The system 100 includes one or more imaging systems 105A, 105B,. . . , 105N (105, in general). The imaging systems 105A, 105B, . . . ,105N are used to acquire images sets 110 a, 110 b, 110 n, respectively(110, in general). The imaging systems 105 can be different orsubstantially similar to one another. For example, the imaging system105A can be a slit-lamp camera and the imaging system 105B can be astandard digital photography camera. In another example, the imagingsystems 105A, 105B, . . . , 105N can all be slit-lamp cameras ofdifferent makes, models or have imaging parameters that are differentfrom one another. The corresponding image sets 110 acquired by thedifferent imaging systems 105 can therefore vary significantly from oneanother. For example, the images across the different imaging systems105 can vary from one another in resolution, white balance, lightingcharacteristics or other image parameters. In such cases, images takenby different imaging systems cannot be reliably compared to one anotherbased simply on visual inspection by a human observer. For example, iftwo eye images taken by different imaging systems are compared by ahuman observer, the perceived difference in conjunctival redness may bedue to a difference in white balance in the two imaging systems thatincorrectly makes one image look redder than the other.

In some implementations, the images 110 acquired by the same imagingsystem 105 can vary from one another. For example, if the images aretaken some time apart, variability due to, for example, parameter driftor different lighting conditions can contribute to the variability ofthe images.

The system 100 includes a color score calculator module 115 that can beused to determine or assign a color score to the images 110 or toportions thereof. The color score calculator module 115 can beimplemented on a computing device and configured to account forvariability that exists in images acquired using one or more imagingsystems 105. In some implementations, the color score calculator module115 can be implemented using a general purpose computer such as adesktop or laptop computer or a mobile device that is capable ofexecuting one or more software programs. In some implementations, thecolor score calculator module 115 is configured to execute one or moreimage processing application programs such as ImageJ, developed at theNational Institutes of Health.

In some implementations, the color score calculator module 115 includesa user interface 118 that is configured to accept user input as well asprovide color score outputs to a user. In some implementations, a usercan interact with the color score calculator module 115 through the userinterface 118. For example, for a given image, the user can select anarea of interest over which the color score is calculated. The userinterface 118 therefore provides the flexibility of a user choosing aportion of an image (rather than the entire image) over which the colorscore is calculated. This is advantageous, for example, in eye imageswhere the white scleral/conjunctival region can be selectively chosenover the corneal region in calculating conjunctival redness.

In some implementations, the color score calculator module 115calculates a color score for the selected region of interest inaccordance with one or more image analysis algorithms described below.The image analysis algorithms can include determining color informationfrom pixels of the selected region of interest and/or other portions ofthe image being analyzed. In general, the color score calculator module115 assigns color scores to the images 110 or portions thereof andoutputs an image set 120 in which each image is associated with astandardized score for a particular color. For example, in case of eyeimages, the image set 120 can include one or more images 110 that areassigned a corresponding CRI. The images from the set 120 and anassociation with the respective color scores can be stored in a storagedevice.

The methods and systems described herein process digital images orportions thereof based on their color properties. Color properties canbe described, for example, using color spaces that represent colors astuples of numbers, typically as three or four values or colorcomponents. Examples of color spaces include RGB, CMY, CMYK, YIQ, YUV,YCrCb, HSV, HSI, IHC and HSL color spaces. In general, color spaces canbe broadly classified into Cartesian and polar coordinate based colorspaces. An understanding of such color spaces is important in themethods and systems described herein and are therefore described nextwith reference to FIGS. 2A-2D.

Referring now to FIG. 2A, an RGB color space is shown as an example of aCartesian color space. In this color space, a color is represented in athree dimensional Cartesian space composed on three colors red, greenand blue. The RGB color space is an additive color model in which red,green, and blue colors are added together in various ways to reproduce abroad array of colors. The RGB color space is typically used forsensing, representation, and display of images in electronic systems,such as televisions, digital cameras, computers and handheld mobiledevices. In the example shown in FIG. 2A, different colors are encodedusing three 8-bit unsigned integers (0 through 255) representing theintensities of red, green, and blue. This representation is the currentmainstream standard representation in image file formats such as JPEG orTIFF. Such encoding of the RGB space results in more than 16 milliondifferent possible colors. As shown in FIG. 2A, the colors at thevertices of the RGB color space may be represented as the followingpoints: (0, 0, 0) is black, (255, 255, 255) is white, (255, 0, 0) isred, (0, 255, 0) is green, (0, 0, 255) is blue, (255, 255, 0) is yellow,(0, 255, 255) is cyan and (255, 0, 255) is magenta. Any point in thevolume bounded by these vertices represents a mixed color that can bebroken down into red, green and blue components and represented in theRGB space as a point (r, g, b). Further, lines and planes may also bedefined in the RGB color space. For example, the line connecting pureblack (0, 0, 0) and pure white (255, 255, 255) may be defined as a grayline 205. Other examples of Cartesian color spaces include the YIQ, YUVand YCbCr spaces.

The Cartesian color spaces, while ideal for describing colors in digitalformats, are not well suited for describing colors that are practicalfor human interpretation. For example, human beings do not perceive acolor in terms of its component primary colors. Rather, humans usuallydescribe a color by its hue, saturation and brightness or intensity. Hueis an attribute that describes what a color actually is (for example,red, yellow, orange, cyan etc.), whereas saturation is a measure thatdescribes to what extent the color is diluted by white light. Brightnessis a descriptor that embodies the achromatic notion of intensity and isan important factor in describing color perception. Color spaces basedon these attributes of colors are ideal for algorithms related to humanperception of color, such as described herein. The IHC (Intensity, Hue,Chroma) color space described with respect to FIG. 2B is an example ofsuch a color space.

Referring to FIG. 2B, the IHC color space includes of a verticalintensity axis 215 and loci 220 a, 220 b (220 in general) of colorpoints that lie on planes perpendicular to the axis. The hue (H) 225 ofa color point within a locus plane (220 a for example) is represented byan angle with respect to a reference point while a chroma (C) 230 isrepresented as a linear distance of the point from the point ofintersection of the locus plane 220 a with the intensity axis 215. Eventhough, the example in FIG. 2B shows the loci 220 to be circular inshape, other polygonal shapes, such as triangles, pentagons, hexagonsetc., may be used to represent the loci. The area of the loci 220 is afunction of the intensity. In other words, the range of chroma is alsodependent on the intensity. For example, at zero intensity (i.e. I=0),all colors have zero chroma value and converge to black. Similarly, forthe maximum intensity (e.g. I=1), all colors have zero chroma value andconverge to white. Within this range, the area of the loci 220 (or therange of chroma values) may increase, for example from I=0 to I=0.5 andthen decrease again from I=0.5 to I=1. FIG. 2B shows the locus 220 bcorresponding to intensity I=0.75. For a given locus plane 220, the hueof a color point is determined by an angle from a reference point. Inthis example, red designates the reference point, i.e. zero hue, and thehue increases in a counterclockwise direction from the reference point.Other polar coordinate based color spaces, such as the HSL (Hue,Saturation, Lightness) and HSV (Hue, Saturation, Value) color spaces,also follow similar principles with hue being represented as an angle inan polar coordinate based coordinate system.

Referring now to FIG. 2C, a HSL color space also includes of a verticalaxis and loci 220 of color points that lie on planes perpendicular tothe axis. In this color space, the vertical axis represents lightness(L) 234. The HSL color space is also referred to HLS or HSI with Istanding for intensity. The HSL color space represents colors as pointsin a cylinder 231 (called a color solid) whose central axis 234 rangesfrom black at the bottom to white at the top, with colors distributedbetween these two extremities. The angle around the axis corresponds tothe hue 225, the distance of a point on a given locus 220 from the axiscorresponds to the saturation 232, and the distance along the axis 234corresponds to lightness or intensity. Unlike the chroma 230 in the IHCcolor space (FIG. 2A), the range of the saturation 232 is not a functionof the lightness or intensity.

Referring now to FIG. 2D, an example of a HSV color space representscolors via an inverted color cone 238 on a cylinder 240. Otherrepresentations of the HSV color space are also possible. In thisexample, the HSV color space includes a common vertical axis 236 for thecone 238 and the cylinder 240. The central axis 236 ranges from black atthe bottom to white at the top, with colors represented in loci 220distributed between these two extremities. The angle around the axiscorresponds to the hue 225, the distance of a point on a given locus 220from the axis corresponds to the saturation 232, and the distance alongthe axis 234 corresponds to the value V. The value can be scaled to bebetween 0 and 1. In this color space, the saturation 232 is a functionof the value V when V is between 0.5 and 1. For example, when V=1, allcolors converge to pure white. When V is between, 0 and 0.5, the rangeof the saturation is constant and not a function of the value, as shownin FIG. 2D.

In some implementations, hue information from digital images are used inthe methods and systems described herein. In some implementations, colorinformation corresponding to pixels in a digital image are converted toa polar coordinate based color space in determining a color score thatrepresents a particular color content. For example, in determining aredness value for a portion of a digital eye image, the colorinformation from the pixels can be converted from the RGB color space tothe HSV color space and the hue information can be used in calculatingthe redness score of the portion. As described with respect to FIG. 2B,in general, hue is an attribute of polar coordinate based color spaceswhile most digital images are represented using Cartesian coordinatesystems such as the RGB color model. The RGB color information may betransformed into a polar coordinate based color space such as the HSIcolor space. For example, the hue may be calculated as:

$H = \left\{ {{\begin{matrix}\theta & {B \leq G} \\{{360 - \theta},} & {B > G}\end{matrix}{where}\mspace{14mu}\theta} = {\cos^{- 1}\left\{ \frac{\frac{1}{2}\left\lbrack {\left( {R - G} \right) + \left( {R - B} \right)} \right\rbrack}{\left\lbrack {\left( {R - G} \right)^{2} + {\left( {R - B} \right)\left( {G - B} \right)}} \right\rbrack^{1/2}} \right\}}} \right.$

The saturation component is given by:

$S = {1 - {\frac{3}{\left( {R + G + B} \right)}\left\lbrack {\min\left( {R,G,B} \right)} \right\rbrack}}$

The intensity of the component is given by:

$I = {\frac{1}{3}\left( {R + G + B} \right)}$

In some implementations, the RGB color information can be transformedinto the HSV color space using the following equations. For example, thevalue component V can be calculated as:V=max(R,G,B)

The saturation component S can be calculated as:

$S = {\frac{delta}{\max\left( {R,G,B} \right)}\left\{ \begin{matrix}{if} & {{\max\left( {R,G,B} \right)} \neq 0} \\{else} & {S = 0}\end{matrix} \right.}$

whereindelta=max(R,G,B)−min(R,G,B)

The hue component H is given by:

$\quad\left\{ \begin{matrix}{{delta} \neq 0} & \left\{ \begin{matrix}{H = {\frac{{60 \times \left( \frac{G - B}{delta} \right)} + 360}{360}\left\{ {{{if}\mspace{14mu}{\max\left( {R,G,B} \right)}} = R} \right\}}} \\{H = {\frac{{60 \times \left( \frac{B - R}{delta} \right)} + 360}{360}\left\{ {{if}\mspace{14mu}{\max\left( {R,G,{B = G}} \right\}}} \right.}} \\{H = {\frac{{60 \times \left( \frac{R - G}{delta} \right)} + 360}{360}\left\{ {otherwise} \right\}}}\end{matrix} \right. \\{{delta} = 0} & \left\{ {H = 0} \right\}\end{matrix} \right.$

Referring now to FIG. 3, a flowchart 300 represents an example sequenceof operations for determining a color score of a digital image. In someimplementations one or more of the operations can be executed at thecolor score calculator module 115 described with reference to FIG. 1.

The operations include obtaining a digital image of biological tissue(302). The digital image can be obtained from an imaging systemsubstantially similar to the imaging system 105 described with referenceto FIG. 1. In some implementations, the digital image can be obtainedsubstantially directly from the imaging system. In some implementations,obtaining the digital image can include retrieving the digital imagefrom a storage device. The digital image can include, for example, animage of a human eye wherein the biological tissue is an ocular tissue(for example, conjunctiva). The biological tissue can also be anepidermal tissue (for example, facial epidermis).

Operations can also include receiving a selection of an evaluation areain the image (304). The selection can be received, for example, througha user interface substantially similar to the user interface 118described with reference to FIG. 1. The selection of the evaluation area(or region of interest) allows a user, in a semi-automated process, toidentify a subset of pixels from the digital image for which the colorscore is calculated. This allows for excluding pixels that are notconsidered relevant or should be excluded in calculating the colorscore. For example, in case of determining conjunctival redness from aneye image, the corneal region may be considered irrelevant. In suchcases, a user can select the white scleral/conjunctival region from theeye image to be considered in the color score calculation. The selectionof the evaluation area also allows a user to modify a shape and/or sizeof the evaluation as needed. For example, if an eye image shows a redspot in a small portion of the scleral/conjunctival region, calculatingthe color score based on a small area that just includes the spot canyield a misleadingly high score. However, by including the entireconjunctival area in the color score calculation, a more meaningfulscore can be obtained. In some implementations, a very specific area canbe precisely selected in the image for evaluation, thereby obtaining ascore more representative of the selected area than to the completeimage. This interactive semi-automatic process also allows forrepeatability to evaluate a substantially same area in different images.

The selection of the evaluation area can be implemented using a digitalselector functionality of the user interface. In some implementations,the selector function can allow for selecting the evaluation area aspredetermined shapes (e.g. circles or rectangles) of adjustable size. Insome implementations, the digital selector can allow for selecting morecomplex shapes such as an irregular polygon. In some implementations, amulti-point selector can allow for selecting any random shapes withinthe digital image. For example, for measuring a color score related toeyelid and skin telangiectasia images, the evaluation area can beselected as oval. In another example, for measuring a color score inimages related to fluorescein stained punctuate keratitis, theevaluation area can be circular. In yet another example, in calculatinga color score for images representing corneal or irisneovascularization, the evaluation area can be circular (e.g. to selectonly the cornea or the iris) or annular. An annular evaluation area canbe selected, for example, to exclude the pupil. In such cases, pupilconstricting agents can be administered prior to acquisition of theimages to minimize variability in pupil diameter. In someimplementations, the color score calculation can be based only on pixelswithin the selected evaluation area.

For each of a plurality of pixels within the evaluation area, theCartesian color components are determined (306). For example, if thedigital image is represented using the RGB color space, the red, greenand blue components corresponding to the pixel value are determined. Insome implementations, the color components can be associated withanother Cartesian color space such as the CMY color space. In someimplementations, the plurality of pixels includes all pixels in theevaluation area. In some implementations, only a subset of the pixelswithin the evaluation is considered in calculating the color score.

Operations further include determining a hue value from the Cartesiancolor components (308). As described above with reference to FIGS.2B-2D, hue is a component of a polar coordinate based color space andtherefore determining the hue value can include converting the Cartesiancolor components to a polar coordinate based color space. The polarcoordinate based color space can include, for example, the HSV colorspace. Converting the Cartesian color components to a polar co-ordinatebased color space can be done, for example, using the conversionsdescribed above with reference to FIGS. 2A-2D. The hue value can bedescribed in terms of an angle also as described with reference to FIGS.2B-2D.

Operations also include determining a color value for each of theplurality of pixels (310). In some implementations, determining thecolor value includes mapping an angle corresponding to the hue value toa scalar value within a predetermined range. For example, in determiningconjunctival redness, the hue values corresponding to the color red maylie within −60° and 60°. In some implementations, the angle values canbe mapped to a scalar range between, for example, 0 and 1. The mappingmay be linear or non-linear. FIG. 3B shows an example of using anon-linear curve such as a parabola 320 to map hue values between −60°and 60° (represented along the x-axis) to a scalar range between 0 and 1(represented along the y-axis). Other angle ranges can also be used. Insome implementations, the angle range that is mapped on to the scalarvalue can be selected based on a location of a color in the hue circle.For example, in measuring a color score for images related tofluorescein stained punctuate keratitis, an angle range corresponding toobserved shades of green (or green-yellow) can be selected. In anotherexample, for images related to conjunctival/scleral pigmented lesions,an angle range corresponding to yellow can be selected. Even for thesame color, the selected angle range can vary from one type of image toanother. For example, the angle range that is selected for scoringconjunctival redness can be different from the range selected to scoreredness in eyelid and skin telangiectasia images.

In some implementations, the scalar value itself can be taken as thecolor value. In some implementations, the color value for a given pixelis determined as a product of the scalar value and one or morecomponents of the polar coordinate based color space. In someimplementations, when the HSV color space is used, the color value canbe determined as a product of the scalar value with one or both of the Sand V components. For example, in scoring conjunctival redness, thecolor value can be determined as a product of the scalar value and the Scomponent only, whereas in scoring conjunctival/scleral pigmentedlesions or fluorescein stained punctuate keratitis images, the scalarvalue can be multiplied with both the S and V components.

Operations further include assigning a color score to the evaluationarea (312). In some implementations, the color score is determined as anaverage of the color values corresponding to the plurality of pixels forwhich the color values are computed. In some implementations, othermeasures of central tendency such as weighted average, median value ormode can also be used in determining the color score. In someimplementations, the color score is scaled to a value within apredetermined range (e.g. [0, 100]) before being assigned to anevaluation area. The predetermined range can be chosen based on, forexample, the type of image or application. For example, thepredetermined range for conjunctival redness can be different from therange associated with corneal neovascularization. In someimplementations, the scaling can be such that the highest determinedcolor value maps on to the upper end of the predetermined range (100, inthis example) and the lowest determined color value maps on to the lowerend of the predetermined range (0, in this example). The color score isthen mapped on to an appropriate value within the predetermined range.In some implementations, the predetermined range can be fixed based onpredetermined high and low color values. In such cases, color valueshigher than the highest predetermined value are mapped on to the upperend of the range and color values lower than the lowest predeterminedvalue are mapped on to the lower end of the range. In someimplementations, the determined color score is saved in a storage devicealong with an association that links the score with the correspondingimage.

In some implementations, the obtained digital image may be subjected toone or more pre-processing operations prior to calculating the colorscore. The pre-processing operations can include, for example, noiseremoval or white balancing. In some implementations, the obtaineddigital image includes an area of a reference white that is used for thewhite balance operation. The area of the reference white can be selectedfrom an area of biological tissue represented in the image. This isillustrated in the example shown in FIGS. 3C and 3D. FIG. 3C shows anacquired image before white balancing, and FIG. 3D shows a correspondingwhite-balanced image. In this example, a spot in the image 3C is chosenas the reference white and the white balancing is performed based on thereference. In some implementations, the area of reference white isselected from a representation of an external marker (e.g., a whitebackground, or a white strip placed in the imaged region) within theimage. FIGS. 3E-3F illustrate examples of such use of an externalmarker. FIG. 3E shows an image that is captured in the presence of anexternal marker 330. FIG. 3F shows the corresponding white-balancedimage wherein the white balancing is performed using a portion of themarker 330 as the reference white. FIGS. 3G-3L show other examples ofsimilar external marker based white balancing, where FIGS. 3G, 3I, and3K represent the original images, and FIGS. 3H, 3J, and 3L represent thecorresponding white balanced versions, respectively. White balancingallows for a gain adjustment across images acquired using differentimaging systems and/or at different times or lighting conditions.

FIG. 4 is a flowchart representing an example sequence of operations fornormalizing gain in a digital image. In some implementations, at leastsome of the operations depicted in FIG. 4 are executed in the colorscore calculation module 115 described with reference to FIG. 1. Theoperations can also be performed in a separate preprocessing module.

Operations can include receiving a selection of at least a portion of areference white color in a digital image (402). In some implementations,a user can manually select the reference white area through a userinterface such as the user interface 118 described with reference toFIG. 1. The white reference area can be selected by a digital selectortool substantially similar to the digital selector described withreference to FIG. 3A. In some implementations, the white reference canbe automatically selected. For example, if a reference white marker ispositioned such that the marker always appears in a predeterminedposition (e.g., upper left hand corner) within an image, the pixelscorresponding to the predetermined position can be automaticallyselected as representing the reference white.

In some implementations, areas with specular reflection are avoided inselecting the white reference. For the conjunctival case, all the areaswith specular reflection (the red, green and blue values are all above220) were discarded in the redness evaluation. In some implementations,another color can be used as a reference. For example, in determiningcolor score to determine a degree of corneal neovascularization, anon-vascularized area can be selected as the reference.

Operations can also include determining an average gain associated withpixels within the reference white area (404). Determining an averagegain can include, for example, calculating the average of the colorcomponents of the Cartesian color space in which the digital image isrepresented. For example, if the digital image is represented using theRGB color space, calculating the average gain can include determiningthe average red, green and blue components for a plurality of pixelswithin the reference white area. The plurality of pixels can include allor a subset of pixels from the reference white area.

Determining the average gain also includes converting the averageCartesian components (e.g., average red, green and blue values) into acorresponding polar coordinate based representation. In someimplementations, the polar coordinate based color space is the HSV colorspace.

From the polar coordinate based color space representation of theaverage color components, the average gain for the image is determined.For example, in the HSV color space the V value corresponding to theaverage of the color components represents an average gain of the image.

Operations can further include applying the average gain to a pluralityof pixels from the digital image (406). In some implementations, theaverage gain is applied for all pixels within the digital image.Alternatively, the average gain can be applied only to pixels within theregion of interest that is considered for calculating the color score.Such gain adjustment allows normalizing of images from various sourcesand/or that are acquired under different conditions. Even though FIG. 4describes gain adjustment via a white balance operation, it should berecognized that other color balancing operations such as gray balance orneutral balance are also within the scope of this disclosure.

FIG. 5 is a schematic diagram of a computer system 500. The system 500can be used for the operations described with reference to theflowcharts in FIGS. 3A and 4. The system 500 can be incorporated invarious computing devices such as a desktop computer 501, server 502,and/or a mobile device 503 such as a laptop computer, mobile phone,tablet computer or electronic reader device. The system 500 includes aprocessor 510, a memory 520, a storage device 530, and an input/outputdevice 540. Each of the components 510, 520, 530, and 540 areinterconnected using a system bus 550. The processor 510 is capable ofprocessing instructions for execution within the system 500. In oneimplementation, the processor 510 is a single-threaded processor. Inanother implementation, the processor 510 is a multi-threaded processor.The processor 510 is capable of processing instructions stored in thememory 520 or on the storage device 530 to display graphical informationfor a user interface on the input/output device 540.

The memory 520 stores information within the system 500. In someimplementations, the memory 520 is a computer-readable storage medium.The memory 520 can include volatile memory and/or non-volatile memory.The storage device 530 is capable of providing mass storage for thesystem 500. In one implementation, the storage device 530 is acomputer-readable medium. In various different implementations, thestorage device 530 may be a floppy disk device, a hard disk device, anoptical disk device, or a tape device.

The input/output device 540 provides input/output operations for thesystem 500. In some implementations, the input/output device 540includes a keyboard and/or pointing device. In some implementations, theinput/output device 540 includes a display unit for displaying graphicaluser interfaces. In some implementations the input/output device can beconfigured to accept verbal (e.g. spoken) inputs.

The features described can be implemented in digital electroniccircuitry, or in computer hardware, firmware, or in combinations ofthese. The features can be implemented in a computer program producttangibly embodied in an information carrier, e.g., in a machine-readablestorage device, for execution by a programmable processor; and featurescan be performed by a programmable processor executing a program ofinstructions to perform functions of the described implementations byoperating on input data and generating output. The described featurescan be implemented in one or more computer programs that are executableon a programmable system including at least one programmable processorcoupled to receive data and instructions from, and to transmit data andinstructions to, a data storage system, at least one input device, andat least one output device. A computer program includes a set ofinstructions that can be used, directly or indirectly, in a computer toperform a certain activity or bring about a certain result. A computerprogram can be written in any form of programming language, includingcompiled or interpreted languages, and it can be deployed in any form,including as a stand-alone program or as a module, component,subroutine, or other unit suitable for use in a computing environment.

Suitable processors for the execution of a program of instructionsinclude, by way of example, both general and special purposemicroprocessors, and the sole processor or one of multiple processors ofany kind of computer. Generally, a processor will receive instructionsand data from a read-only memory or a random access memory or both.Computers include a processor for executing instructions and one or morememories for storing instructions and data. Generally, a computer willalso include, or be operatively coupled to communicate with, one or moremass storage devices for storing data files; such devices includemagnetic disks, such as internal hard disks and removable disks;magneto-optical disks; and optical disks. Storage devices suitable fortangibly embodying computer program instructions and data include allforms of non-volatile memory, including by way of example semiconductormemory devices, such as EPROM, EEPROM, and flash memory devices;magnetic disks such as internal hard disks and removable disks;magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor andthe memory can be supplemented by, or incorporated in, ASICs(application-specific integrated circuits).

To provide for interaction with a user, the features can be implementedon a computer having a display device such as a CRT (cathode ray tube),LCD (liquid crystal display) monitor, eInk display or another type ofdisplay for displaying information to the user and a keyboard and apointing device such as a mouse or a trackball by which the user canprovide input to the computer.

The features can be implemented in a computer system that includes aback-end component, such as a data server, or that includes a middlewarecomponent, such as an application server or an Internet server, or thatincludes a front-end component, such as a client computer having agraphical user interface or an Internet browser, or any combination ofthem. The components of the system can be connected by any form ormedium of digital data communication such as a communication network.Examples of communication networks include, e.g., a LAN, a WAN, and thecomputers and networks forming the Internet.

The computer system can include clients and servers. A client and serverare generally remote from each other and typically interact through anetwork, such as the described one. The relationship of client andserver arises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other.

The processor 510 carries out instructions related to a computerprogram. The processor 510 may include hardware such as logic gates,adders, multipliers and counters. The processor 510 may further includea separate arithmetic logic unit (ALU) that performs arithmetic andlogical operations.

The methods and systems described herein can be used in a number ofclinical applications. For example, a color score can be used todetermine the severity of a disorder associated with the presence ofthat color. Taking conjunctival redness as an example, the presence ofwhich is associated with a number of conditions (including, but notlimited to dry eye syndrome, conjunctivitis, subconjunctival hemorrhage,blepharitis, acute glaucoma, allergy, injury, keratitis, iritis,episcleritis, scleritis, uveitis, inflamed pterygium, inflamedpinguecula, airborne contaminants, tick borne illnesses like RockyMountain spotted fever, high stress levels and drug use includingcannabis), a higher red color score (e.g., Conjunctival Redness Index(CRI)) determined by a method described herein is associated withgreater severity of the condition. A lower score indicates that thecondition is less severe).

The methods can also be used to monitor progression or treatment of acondition. For example, a first color score is determined at a firsttime point, e.g., before or during administration of a treatment for thecondition associated with the presence of the color, and a second colorscore is determined at a later time. Again taking conjunctival rednessas an example, a first red color score (e.g., CRI) is determined at afirst time point, e.g., before or during treatment, and a second redcolor score is determined at a later time point. The two scores are thencompared, and an increase in the color score indicates progression(i.e., worsening) of the condition or a lack of efficacy of thetreatment; no change indicates that any treatment has at best stoppedprogression (in a progressive disorder), or has been ineffective; and adecrease in the color score indicates that the treatment has beeneffective. One of skill in the art will appreciate that the treatmentwill vary depending on the exact condition; common treatments includethe application of cold or hot compresses; gentle washing; andadministration of topical and/or systemic antibiotics,anti-inflammatories, or steroids.

EXAMPLES

The methods and systems described herein are further described using thefollowing examples (with reference to FIGS. 6A-6C), which do not limitthe scope of the claims. FIGS. 6A-6C illustrate color scoredetermination for conjunctival redness. The images were obtained frompatients with ocular surface diseases such as dry eye disease. AHaag-Streit BQ 900 IM imaging system was used to acquire the images. Theimages were acquired using the same exposure and lighting parameters.However, because of the particular anatomic characteristics of eachindividual, angulation of the light source was customized in each caseto provide good illumination and reduce confocal reflection over thearea of interest.

The nasal conjunctiva of either left or right eye was captured whilepatients looked to the extreme ipsilateral side of the photographed eye,i.e., extreme right for right eyes or extreme left for left eyes. Imageswith different degrees of conjunctival redness were included in theexperiment of determining the color score. Two clinicians evaluated theimages and independently graded conjunctival redness based on twowell-known image-based scales, the Efron (Efron, Optician. 213:26-35(1997); Efron, Optician 219:44-45 (2000)) and Validated Bulbar Redness(VBR; Schulze et al., Optom Vis Sci. 2007; 84:976-983; see also Schulzeet al., Invest. Ophthalmol. Vis. Sci. 52(8):5812-5817 (2011)) scales.Conjunctival redness for the same images was also evaluated using themethods and systems described herein. For example, in FIG. 6A, theregion 610 was selected as the evaluation area using the digitalselector curve 620. For the image shown in FIG. 6B, the region 630 wasselected as the evaluation area using the digital selector curve 640.Similarly, in FIG. 6C, the region 650 was selected as the evaluationarea using the digital selector curve 660.

The algorithm was implemented on the Java-based imaging-processingplatform ImageJ (National Institutes of Health; Rasband, ImageJ, U.S.National Institutes of Health, Bethesda, Md., USA (imagej.nih.gov/ij/),1997-2011; Abramoff et al., Biophotonics International (11)7:36-42(2004)) as two plugins, one for white balancing and other for rednessquantification. All images were exported from the slit-lamp camera ofthe imaging system as TIFF files to a personal computer executing ImageJand the above mentioned plug-ins.

In these examples, because no reference white markers (strips) were usedduring image acquisition, a white spot from a white area in each of theimages was selected as the reference for the white-balancing (avoidinghyper-white areas such as the spot 635 in FIG. 6B caused by confocalreflection of the light source).

The digital selector tool used for these experiments included sevenadjustable points that were selected using a mouse pointer. The selectortool included left or right options depending on the side of the eyethat was evaluated. The selector tool was used to select the evaluationarea as the exposed nasal or temporal conjunctiva visible in therespective images. In general, for all images, the conjunctival area wasselected as the area of interest excluding the cornea, lids or eyelashesfrom the selected evaluation area.

After all the images were scored, a table with all the scores wasexported to a spreadsheet for analysis. The scores were also displayedon the images. For example, a score display 625 on the image shown inFIG. 6A indicated a redness score of 10.90. Similarly the image shown inFIG. 6B received a score of 61.07 and the image shown in FIG. 6Creceived a score of 97.01. The redness scores were then correlated withthe corresponding subjective scores from the two clinical examiners foreach one of the two subjective grading scales. The results for the Efronscale from the two clinical examiners are shown in FIGS. 7A and 7B,respectively. The results for the VBR scale from the two clinicalexaminers are shown in FIGS. 7C and 7D, respectively. FIG. 7A shows aplot in which each evaluated photograph (represented by a dot 710) isrepresented in a coordinate system where the y-axis represents the Efronvalues assigned by the clinical examiner and the x-axis represents theCRI values. The plot 720 represents a regression line fitted on thedata. A metric known as the Spearman's coefficient of correlation (R)was computed based on the data. The coefficient of correlation is ametric that represents the degree of co-variation between two variables,and indicates the degree to which two variable's movements areassociated. The value of R varies between −1.0 and 1.0. An R value of1.0 indicates that there is perfect correlation between the scores ofboth variables. An R value of −1.0 indicates perfect correlation in themagnitude of the scores of both variables but in the opposite direction.On the other hand, an R value of 0 indicates that there is nocorrelation between the scores of the two variables. For the data setrepresented in FIG. 7A, the R value was 0.925, thus indicating that theCRI values were strongly correlated to the Efron value evaluations ofthe first clinical examiner. FIG. 7B represents the Efron valueevaluation data from the second clinical examiner and in this case the Rvalue was 0.857. FIG. 7C represents the VBR evaluation data from thefirst clinical examiner and in this case the R value was 0.830. FIG. 7Drepresents the VBR evaluation data from the second clinical examiner andin this case the R value was 0821. All of the above correlations werecomputed to be statistically significant (P<0.001). The high values ofthe coefficient of correlation for all the cases indicate that a strongassociation was found between the computer derived analysis and thesubjective clinical evaluations.

Other Embodiments

A number of implementations have been described. Nevertheless, it willbe understood that various modifications may be made. For example,elements of one or more implementations may be combined, deleted,modified, or supplemented to form further implementations. As yetanother example, the logic flows depicted in the figures do not requirethe particular order shown, or sequential order, to achieve desirableresults. In addition, other steps may be provided, or steps may beeliminated, from the described flows, and other components may be addedto, or removed from, the described systems. Accordingly, otherimplementations are within the scope of the following claims.

What is claimed is:
 1. A computer implemented method for evaluating acondition of an ocular surface based on a color of the ocular surface,the method comprising: obtaining a digital signal representing at leasta portion of the ocular surface; determining, from a portion of thedigital signal, a plurality of color components that are based on aCartesian color space; determining, from the color components, a huevalue in a polar coordinate based color space; selecting an angle rangein the polar coordinate based color space in accordance with a colorassociated with the condition being evaluated; determining that the huevalue lies within the angle range that represents the color associatedwith the condition being evaluated in a hue circle representation of thepolar coordinate based color space; mapping the hue value to a scalarvalue within a predetermined range in response to determining that thehue value lies within the angle range; selecting a function of one orboth of two other components of the polar coordinate based color space,wherein the function is selected based on the condition being evaluated;determining a color value as a product of the scalar value and thefunction of one or both of the two other components of the polarcoordinate based color space; and generating a color score as a functionof the color value, wherein the color score of the ocular surfacerelates to a quantitative evaluation of the condition of the ocularsurface.
 2. The method of claim 1, further comprising receiving aselection of at least a first portion of the digital signal; determiningan average gain associated with the first portion of the digital signal;and applying the average gain to a second portion of the digital signal.3. The method of claim 1, wherein mapping the hue value to the scalarvalue comprises using a parabolic curve to map the hue value to thescalar value within the predetermined range.
 4. The method of claim 1,further comprising receiving a selection of a portion of the digitalsignal through a graphical user interface, wherein assigning the colorscore comprises assigning the color score to the portion of the digitalsignal.
 5. The method of claim 1, wherein the color score indicates adegree of redness of the ocular surface.
 6. The method of claim 1,wherein the ocular surface comprises conjunctiva.
 7. The method of claim1, wherein the Cartesian color space is an RGB color space.
 8. Themethod of claim 1, wherein the polar coordinate based color space is anHSV color space.
 9. The method of claim 1, further comprising storingthe color score and an association of the color score with the digitalsignal.
 10. The method of claim 1, wherein the digital signal is relatedto one or more of corneal neovascularization, fluorescein stainedepithelial punctate keratitis and epithelial defects, eyelid and skintelangiectasia, and conjunctival/scleral pigmented lesions.
 11. Acomputer readable storage device having encoded thereon computerreadable instructions, which when executed by a processor, cause aprocessor to perform operations comprising: obtaining a digital signalrepresenting at least a portion of an ocular surface; determining, froma portion of the digital signal, a plurality of color components thatare based on a Cartesian color space; determining, from the colorcomponents, a hue value in a polar coordinate based color space;selecting an angle range in the polar coordinate based color space inaccordance with a color associated with a condition of the ocularsurface; determining that the hue value lies within the angle range thatrepresents the color associated with the condition being evaluated in ahue circle representation of the polar coordinate based color space;mapping the hue value to a scalar value within a predetermined range inresponse to determining that the hue value lies within the angle range;selecting a function of one or both of two other components of the polarcoordinate based color space, wherein the function is selected based onthe condition being evaluated; determining a color value as a product ofthe scalar value and the function of one or both of the two othercomponents of the polar coordinate based color space; and generating acolor score as a function of the color value, wherein the color score ofthe ocular surface relates to a quantitative evaluation of the conditionof the ocular surface.
 12. The computer readable storage device of claim11, further comprising receiving a selection of at least a first portionof the digital signal; determining an average gain associated with thefirst portion of the digital signal; and applying the average gain to asecond portion of the digital signal.
 13. The computer readable storagedevice of claim 11, wherein mapping the hue value to the scalar valuecomprises using a parabolic curve to map the hue value to the scalarvalue within the predetermined range.
 14. The computer readable storagedevice of claim 11, wherein the operations further comprise receiving aselection of a portion of the digital signal through a graphical userinterface, wherein assigning the color score comprises assigning thecolor score to the portion of the digital signal.
 15. The computerreadable storage device of claim 11, wherein the color score indicates adegree of redness of the ocular surface.
 16. The computer readablestorage device of claim 11, wherein the ocular surface comprisesconjunctiva.
 17. The computer readable storage device of claim 11,wherein the Cartesian color space is an RGB color space.
 18. Thecomputer readable storage device of claim 11, wherein the polarcoordinate based color space is an HSV color space.
 19. The computerreadable storage device of claim 11, wherein the operations furthercomprise storing the color score and an association of the color scorewith the digital signal.
 20. The computer readable storage device ofclaim 11, wherein the digital signal is related to one or more ofcorneal neovascularization, fluorescein stained epithelial punctatekeratitis and epithelial defects, eyelid and skin telangiectasia, andconjunctival/scleral pigmented lesions.